Skip to content

Conversation

michaelsproul
Copy link
Member

@michaelsproul michaelsproul commented Jul 17, 2025

Issue Addressed

A different (and complementary) approach for:

Proposed Changes

This PR adds a flag to set the DA boundary to the Deneb fork. The effect of this change is that Lighthouse will try to backfill all blobs.

Most peers do not have this data, but I'm thinking that combined with trusted-peers this could be quite effective.

Additional Info

  • We turn on --prune-blobs false when this flag is enabled, and override any setting of --prune-blobs true.
  • Should we turn on genesis-backfill whenever complete-blob-backfill is turned on? I'm inclined to say no, because users may want to backfill blobs only to e.g. 5 months (standard block backfill distance).

@chong-he
Copy link
Member

chong-he commented Sep 3, 2025

I have done some testing on this together with @michaelsproul

To backfill the blobs beyond the 18-day window, we supply to the node that we want to backfill the following flags:
--libp2p-addresses
--trusted-peer

Example: lighthouse bn --libp2p-addresses /ip4/ip_address/tcp/9000 --trusted-peer 16Ulongstring --some-other-flags

It is also recommended to supply the same flag to the source node (the node with the blobs data).

Some caveats/known issues:

  1. From time to time the following logs will appear:
ERROR Backfill sync failed                          error: BatchDownloadFailed(Epoch(365247))
WARN  Backfill batch failed to download. Penalizing peers  score_adjustment: Low Tolerance Error, batch_epoch: 365329 

I believe this is expected as most of the peers do not have the blobs data outside the 18-day window. As the peers are penalized, this results in a fluctuating peer count, as evident from the connected peer below (note that the default peer is now 200 in the unstable branch):

Screenshot from 2025-09-03 20-47-18
  1. The backfill of blocks+blobs further from the head could be slow, in the range of 2-4 slots/sec (this could also be partly due to hardware limitations). So a complete backfill of blobs until the Deneb fork is estimated to take some time, could be as long as a few weeks.

    This could be improve if there are more nodes with blobs data are supplied - though this is not yet tested. In the testing, we supply only 1 node

  2. The backfill could stuck - I have not investigated this. When it stalls, a restart of the beacon node will "resume" the backfill

Additional info: the node blobs data has backfilled to slot 10.2M from a fresh checkpoint sync:

"anchor": {
    "anchor_slot": "12452832",
    "oldest_block_slot": "10202176",
    "oldest_block_parent": "0xc91bf2f51b1ffdefe6dd791ba9c0465bf486cedea6f28d231b8f5742012be6d9",
    "state_upper_limit": "18446744073709551615",
    "state_lower_limit": "0"
  },
  "blob_info": {
    "oldest_blob_slot": "10202176",
    "blobs_db": true
  }
}

@michaelsproul michaelsproul added ready-for-review The code is ready for review and removed work-in-progress PR is a work-in-progress labels Sep 10, 2025
@michaelsproul michaelsproul added v8.0.0-rc.0 Q3 2025 release for Fusaka on Holesky waiting-on-author The reviewer has suggested changes and awaits thier implementation. and removed ready-for-review The code is ready for review labels Sep 11, 2025
@mergify mergify bot added ready-for-review The code is ready for review and removed waiting-on-author The reviewer has suggested changes and awaits thier implementation. labels Sep 18, 2025
Copy link
Member

@jimmygchen jimmygchen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice, looks good to me. Just need to disable the prune blobs behaviour when the flag is present.

@jimmygchen jimmygchen added waiting-on-author The reviewer has suggested changes and awaits thier implementation. and removed ready-for-review The code is ready for review labels Sep 18, 2025
@michaelsproul
Copy link
Member Author

Done. I opted to just override --prune-blobs true if it is set, as this was simpler than re-parsing the same flag again, or fiddling with the order of parsing. I've added a test that confirms the behaviour, so we can always change it if we want to, but won't change it accidentally.

@michaelsproul michaelsproul added ready-for-review The code is ready for review and removed waiting-on-author The reviewer has suggested changes and awaits thier implementation. labels Sep 18, 2025
Copy link
Member

@jimmygchen jimmygchen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, thanks!

@jimmygchen jimmygchen added ready-for-merge This PR is ready to merge. and removed ready-for-review The code is ready for review labels Sep 18, 2025
mergify bot added a commit that referenced this pull request Sep 18, 2025
mergify bot added a commit that referenced this pull request Sep 18, 2025
mergify bot added a commit that referenced this pull request Sep 18, 2025
mergify bot added a commit that referenced this pull request Sep 18, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
database deneb ready-for-merge This PR is ready to merge. v8.0.0-rc.0 Q3 2025 release for Fusaka on Holesky
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants